Hybrid N-gram Probability Estimation in Morphologically Rich Languages

N-gram language modeling is essential in natural language processing and speech processing. In morphologically rich languages such as Korean, a word usually consists of at least one lemma (content morpheme) and functional morphemes which represent various grammatical. Most word forms in Korean, however, have problems of sparse data and zero probability, because of quite complex morpheme combinations. Thus morpheme-based N-gram modeling is widely used instead of a word sequence modeling. In this paper, we contend that a morpheme-based N-gram is inefficient language modeling in that it inevitably approximates the probability of unnecessary morpheme sequences, so the longer sequences we have, the lower probability estimates we get. We suggest a hybrid method that joins word-based and morpheme-based language modeling. The new method can also be regarded as an extension of a class-based measurement. Our experimental results show that the method produces better probability estimation than the morphemebased measurement.